feat(template): replacing template-zero with source-based template (#321) by guzmud · Pull Request #336 · OpenAEV-Platform/collectors

guzmud · 2026-04-21T08:01:03Z

Proposed changes

Adding/refactoring (depending if feat(template): kickstarting the collector template (#319) #320 is merged) a template for collectors
Replacing the collector vs services split with a collector vs source split (cf. internal documentation for now)
Defining a static custom name for the custom configuration (before it was the $name-of-your-collector configuration, changing for each)

Testing Instructions

Copy-paste the template and create a new collector with it

Related issues

Checklist

I consider the submitted work as finished
I tested the code for its functionality
I wrote test cases for the relevant uses case
I added/update the relevant documentation (either on github or on notion)
Where necessary I refactored code to improve the overall quality
For bug fix -> I implemented a test that covers the bug

Further comments

Nota bene: this branch was forked from #320 (hence the commits related to template zero)

Executive summary from the internal documentation

Collectors can be seen as data processing unit. With this frame of mind, the already existing work pointing out towards the potentiality of a template can be turned into a more actionable template based around the definition of a source fed into a generic collector. This should lead to a new template that could be more dev-friendly, easier to explain, use and maintain.

codecov · 2026-04-28T15:21:11Z

Codecov Report

❌ Patch coverage is 23.92027% with 229 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.00%. Comparing base (03b39fa) to head (cc0c717).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
template/tests/services/conftest.py	3.22%	90 Missing ⚠️
template/tests/services/fixtures/factories.py	8.75%	73 Missing ⚠️
template/tests/test_create_collector.py	9.52%	38 Missing ⚠️
template/tests/conftest.py	28.57%	20 Missing ⚠️
template/src/models/settings/config_loader.py	65.21%	8 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (03b39fa) and HEAD (cc0c717). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (03b39fa) HEAD (cc0c717)

connectors 1 0

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #336       +/-   ##
===========================================
- Coverage   82.43%   69.00%   -13.43%     
===========================================
  Files          41      139       +98     
  Lines        1674     7076     +5402     
===========================================
+ Hits         1380     4883     +3503     
- Misses        294     2193     +1899

Flag	Coverage Δ
connectors	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…tarting point

… features

Copilot

Pull request overview

Introduces a new collector template that models collectors as a generic engine fed by an implementer-provided source (data fetcher + source data + signatures), replacing the prior “collector vs services” shape with a “collector vs source” split.

Changes:

Adds a reusable collector framework (engine, protocols, models, resilient uploaders) plus a minimal runnable template collector entrypoint.
Adds configuration system based on Pydantic Settings (YAML / .env / env var sources) and docker/packaging scaffolding.
Adds a unittest suite covering protocols, models, engine behavior, and uploaders.

Reviewed changes

Copilot reviewed 50 out of 65 changed files in this pull request and generated 23 comments.

Show a summary per file

File	Description
template/.dockerignore	Docker build context exclusions for config/artifacts/caches.
template/.gitignore	Git ignores for config/artifacts/caches.
template/CONTRIBUTING.md	Contributor docs for the template (currently mismatched with new layout).
template/Dockerfile	Container build/run for the template collector.
template/README.md	Template usage/configuration documentation (currently references old layout).
template/docker-compose.yml	Example compose service for running the template collector.
template/manifest-metadata.json	Collector manifest metadata for the template image.
template/pyproject.toml	Poetry/PEP621 config, deps/extras, dev tooling config, entrypoint script.
template/src/init.py	Package exports for `ConfigLoader`.
template/src/main.py	Module entrypoint calling `main()`.
template/src/config.yml.sample	Sample YAML config for the template.
template/src/img/template-logo.png	Collector icon asset placeholder.
template/src/models/init.py	Exports `ConfigLoader`.
template/src/models/settings/init.py	Exports settings model components.
template/src/models/settings/base_settings.py	Base Pydantic Settings config for nested env parsing and immutability.
template/src/models/settings/collector_configs.py	OpenAEV + collector settings models.
template/src/models/settings/config_loader.py	ConfigLoader that merges sources and flattens config for daemon.
template/src/models/settings/template_configs.py	Template-specific settings (key, time window, batch size).
template/src/py.typed	Marks package as typed (PEP 561).
template/src/source/init.py	Source package marker.
template/src/source/template_data_fetcher.py	Placeholder data fetcher implementation.
template/src/source/template_signatures.py	Placeholder supported signature list.
template/src/source/template_source_data.py	Placeholder source data model implementation.
template/src/template_collector.py	Runnable example that wires Source + BaseCollector and starts it.
template/src/collector/init.py	Collector package marker.
template/src/collector/collector.py	`BaseCollector` daemon wrapper that instantiates/configures the engine.
template/src/collector/engines/init.py	Engines package marker.
template/src/collector/engines/basic.py	Generic processing engine (fetch/filter/process expectations, upload results/traces).
template/src/collector/helpers/init.py	Helpers package marker.
template/src/collector/internals/init.py	Internals package marker.
template/src/collector/internals/oaev_uploaders.py	OpenAEV expectation/trace uploaders built on resilient uploader.
template/src/collector/internals/resilient_uploader.py	Generic bulk uploader with fallback to individual uploads.
template/src/collector/models/init.py	Models package marker.
template/src/collector/models/data.py	`OAEVData` and `TraceData` models.
template/src/collector/models/exception.py	Custom exception hierarchy.
template/src/collector/models/expectations.py	Expectation result/trace/summary models and formatting helpers.
template/src/collector/models/source.py	`Source` definition and default `SourceHandler` implementation.
template/src/collector/protocols/init.py	Protocols package marker.
template/src/collector/protocols/data_fetcher.py	Protocol for data fetchers.
template/src/collector/protocols/engine.py	Protocol for collector engines.
template/src/collector/protocols/source_data.py	Protocol for source data models.
template/src/collector/protocols/source_handler.py	Protocol for source handlers.
template/src/collector/types/init.py	Types package marker.
template/src/collector/types/collector.py	Type aliases for collector structures.
template/src/collector/types/internals.py	Type aliases for resilient uploader injection points.
template/src/collector/utils/init.py	Utils package marker.
template/src/collector/utils/retroport_itertools.py	Python 3.11-compatible `batched()` helper.
template/tests/init.py	Tests package marker.
template/tests/test_template_collector.py	Tests wiring of `template_collector.main()`.
template/tests/collector/init.py	Tests subpackage marker.
template/tests/collector/test_collector.py	Tests `BaseCollector` initialization/setup behaviors.
template/tests/collector/engines/init.py	Tests engines package marker.
template/tests/collector/engines/test_basic.py	Tests `BasicCollectorEngine` behaviors and flow.
template/tests/collector/internals/init.py	Tests internals package marker.
template/tests/collector/internals/test_oaev_uploaders.py	Tests expectation/trace uploader behavior.
template/tests/collector/internals/test_resilient_uploader.py	Tests resilient uploader bulk+fallback behavior.
template/tests/collector/models/init.py	Tests models package marker.
template/tests/collector/models/test_data.py	Tests `OAEVData`/`TraceData` validation and formatting.
template/tests/collector/models/test_expectations.py	Tests expectation models behavior.
template/tests/collector/models/test_source.py	Tests `Source` and `SourceHandler` behavior.
template/tests/collector/protocols/test_data_fetcher.py	Tests protocol conformance for data fetchers.
template/tests/collector/protocols/test_engine.py	Tests protocol conformance for engines.
template/tests/collector/protocols/test_source_data.py	Tests protocol conformance for source data.
template/tests/collector/protocols/test_source_handler.py	Tests protocol conformance for source handlers.
template/tests/collector/utils/init.py	Tests utils package marker.
template/tests/collector/utils/test_retroport_itertools.py	Tests retroported `batched()` selection/behavior.

Comments suppressed due to low confidence (2)

template/src/collector/types/collector.py:4

SignatureGroups is defined as list[dict[str, str]], but get_expectation_signature_groups() and match_signature_groups_and_oaevdata() treat signature_groups as a mapping (.items()) from signature type to a list of signature dicts. This mismatch will either break typing (mypy) or lead implementers of the protocol to return the wrong shape. Update SignatureGroups to the actual structure used (e.g., dict[str, list[dict[str, str]]]) and align protocol/implementations accordingly.

from typing import TypeAlias

SignatureGroups: TypeAlias = list[dict[str, str]]

template/pyproject.toml:128

[tool.cmw] icon-path points to src/img/change-me-logo.png, but this template ships src/img/template-logo.png instead. This will break any tooling that relies on tool.cmw metadata for the icon. Update icon-path (or rename the asset) so the referenced file exists.

[tool.cmw]
install-command = "poetry install --extras local"
config-dump-command = "poetry run python -m src --dump-config-schema"
icon-path = "src/img/change-me-logo.png"

+- `--extra current`: Get pyoaev from Git release/current branch
+- `--extra local`: Get pyoaev locally from `../../client-python`
+
+### Development Installation
+
+```bash
+# Development setup with current pyoaev version
+poetry install -E current --with dev,test


+RUN if [[ ${PYOAEV_GIT_BRANCH_OVERRIDE} ]] ; then \
+    echo "Forcing specific version of client-python" && \
+    apk add --no-cache git && \
+    pip install pip3-autoremove && \
+    pip-autoremove pyoaev -y && \


… model

… config

…r and datafetcher

guzmud · 2026-05-18T08:47:38Z

+]
+
+HttpUrlToString = Annotated[HttpUrl, PlainSerializer(str, return_type=str)]
+TimedeltaInSeconds = Annotated[


@Kakudou it seems TimedeltaInSeconds as never been really used (it exists in palo alto XDR, sentinelone and splunk-es but never really used in any case): should be replace the type of period with it or just delete it ? (I guess you were the one that wrote it in the first place in splunk-es, I don't know if it's a leftover to be delete or a WIP unfinished)

…ecture

github-actions Bot added the filigran team Item from the Filigran team. label Apr 21, 2026

guzmud force-pushed the feature/321-rework-template branch 6 times, most recently from 113bce7 to cc0c717 Compare April 28, 2026 15:13

guzmud force-pushed the feature/321-rework-template branch 8 times, most recently from f8264b6 to 02d5277 Compare May 7, 2026 09:48

guzmud changed the base branch from main to release/current May 7, 2026 09:49

guzmud changed the title ~~[template] feat(collector): replacing template-zero with source-based template~~ [template] feat(collector): replacing template-zero with source-based template (#321) May 7, 2026

guzmud force-pushed the feature/321-rework-template branch from 02d5277 to 4cd73ec Compare May 7, 2026 10:14

guzmud added 11 commits May 7, 2026 13:09

[template] feat(collector): copy-pasting the sentinelone probe as a s…

2ef114f

…tarting point

[template] feat(collector): withdrawing SentinelOne specific code and…

2460166

… features

[template] refactor(collector): expectation models moved to base models

fc02033

[template] feat(protocols): adding source-related protocols

d7e1739

[template] refactor(settings): renaming configs to settings

4eab80e

[template] refactor(collector): exception models move to base models

390988d

[template] feat(protocols): adding collector protocols to the template

22a31d4

[template] feat(types): adding custom collector types

a069524

[template] feat(models): adding collector models to the template

4ec8a1d

[template] feat(uploaders): adding resilient uploaders to internals

cf38d6c

[template] feat(utils): adding a retroport of batched for py3.11 support

2839eef

guzmud added 2 commits May 7, 2026 13:09

[template] feat(engine): adding the basic collector engine

4c39c9f

[template] feat(collector): reworking the common collector elements

1caa57c

guzmud force-pushed the feature/321-rework-template branch from 4cd73ec to 1caa57c Compare May 7, 2026 11:10

guzmud requested a review from Copilot May 7, 2026 11:28

Copilot started reviewing on behalf of guzmud May 7, 2026 11:28 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

guzmud added 3 commits May 7, 2026 17:49

[template] fix(collector): fixing various issues

467ae69

[template] refactor(collector): improving code quality

8307439

[template] feat(types): adding BulkData generic type (+fix test)

012d176

guzmud force-pushed the feature/321-rework-template branch from b3dc46b to 012d176 Compare May 11, 2026 07:29

[template] fix(collector): add slugify for collector type

c6eb06a

guzmud force-pushed the feature/321-rework-template branch from 6c1c2d6 to c6eb06a Compare May 11, 2026 07:59

guzmud added 6 commits May 11, 2026 10:44

[template] fix(uploaders): error between results and valid_results

50443be

[template] fix(config): wrong aliases in config_loader

337a7a6

[template] fix(engine): moving try/except to inside the batch loop

b7732ae

[template] fix(models): fixing missing proper default factory in data…

576621b

… model

[template] feat(settings): moving template config to a generic custom…

49e2234

… config

[template] feat(collector): propagating custom config to sourcehandle…

fb2a085

…r and datafetcher

guzmud force-pushed the feature/321-rework-template branch from b61ba00 to fb2a085 Compare May 18, 2026 07:43

guzmud commented May 18, 2026

View reviewed changes

guzmud added 2 commits May 18, 2026 11:47

[template] fix(config): propagating the renaming from template to custom

781e4ed

[template] chore(README): updating the README to match the new archit…

3bf40f2

…ecture

guzmud mentioned this pull request May 20, 2026

test: test the new collector template #364

Open

SamuelHassine changed the title ~~[template] feat(collector): replacing template-zero with source-based template (#321)~~ feat(template): replacing template-zero with source-based template (#321) Jun 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(template): replacing template-zero with source-based template (#321)#336

feat(template): replacing template-zero with source-based template (#321)#336
guzmud wants to merge 25 commits into
release/currentfrom
feature/321-rework-template

guzmud commented Apr 21, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Apr 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guzmud May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

guzmud commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Testing Instructions

Related issues

Checklist

Further comments

Uh oh!

codecov Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guzmud May 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

guzmud commented Apr 21, 2026 •

edited

Loading

codecov Bot commented Apr 28, 2026 •

edited

Loading